Understanding Human-side Impact of Sampling Image Batches in Subjective Attribute Labeling

Capturing human annotators' subjective responses in image annotation has become crucial as vision-based classifiers expand the range of application areas. While there been significant progress interface design general, relatively little research conducted to understand how elicit reliable and cost-efficient when nature task includes a certain level subjectivity. To bridge this gap, we aim different sampling methods batch labeling, that allows annotators label images simultaneously, can impact performances. In particular, developed three strategies forming batches: (1) uncertainty-based labeling (UL) prioritizes classifier predicts with highest uncertainty, (2) certainty-based (CL), reverse strategy UL, (3) random, baseline approach randomly selects images. Although UL CL solely select be labeled from classifier's point view, hypothesized human-side perception performance may also vary depending on strategies. our study, observed participants were able recognize perceived cognitive load across conditions (CL easiest while most difficult). We trade-off between effectiveness more than random) efficiency (UL efficient least efficient). Based results, discuss implications possible future directions labeling.